Consistent Credibility Criteria: Why have them, what are they, and how do you measure them? (abstract)

The National Aeronautics and Space Administration (NASA) has endeavored to develop a consistent method of assuring the credibility of simulation results. Requirements like "accurate enough for a particular use" present problems. How do you measure accuracy? What has to be accurate? What factors are measured? How do you quantify “enough for the particular use?” Which functions are measured? When during the simulated time are they measured? How do you state uncertainty requirements? These are but a few of the specifics that make writing defendable requirements for simulations many orders of magnitude more difficult that writing requirements for other software. NASA-STD-7009 presents a Credibility Assessment Scale (CAS) containing eight criteria grouped into three categories that attempts to quantify factors influencing simulation credibility. This paper identifies these criteria, provides straightforward definitions and introduces typical methods of assessment. NASA Recognizes the Need for Consistent Measures of M&S Credibility The genesis of NASA’s recent emphasis on assuring the validity of simulations and the credibility of simulation studies is the Columbia Accident Investigation Report (CAIB). It called for NASA to “develop, validate, and maintain physics-based computer models to evaluate Thermal Protection System damage from debris impacts. These tools should provide realistic and timely estimates of any impact damage from possible debris from any source that may ultimately impact the Orbiter. Establish impact damage thresholds that trigger responsive corrective action, such as on-orbit inspection and repair, when indicated.” The Renewed Commitment to Excellence, or Diaz Report, broadened the scope beyond the Space Transportation System. Diaz action item four called for NASA to “develop a standard for the development, documentation, and operation of models and simulations (M&S): 1 This paper draws most of its information from NASA-STD-7009. Unless otherwise noted all references are to that standard. 1 http://standards.nasa.gov/released/NASA/NASA_STD_7009_APPROVED_2008_07_11.pdf 2 http://caib.nasa.gov/ 3 http://www.nasa.gov/pdf/55691main_Diaz_020204.pdf • Documentation, configuration management, and quality assurance • Verification and validation, operational data and trending • Tool management, maintenance, and obsolescence • Training requirements • Best practices for user interfaces • User feedback when results appear unrealistic” The Stafford-Covey Report 4 enjoined “formal development, verification and validation, and outside review plans”. It added that “assumptions should be written down and consistently applied.” Finally it required that “sensitivity analysis and careful analysis of uncertainty were to be performed.” These commitments were to be agency wide. The Chief Engineer Memo’s required that the credibility of M&S results is properly conveyed to those making critical decisions, and that analysts should assure that the credibility of M&S meets the project requirements. NASA was to establish M&S requirements and recommendations that will form a strong foundation for disciplined (structure, management, control) development, validation and use of M&S within NASA and its contractor community, include a standard method to assess the credibility of the M&S presented to the decision maker when making critical decisions (i.e., decisions that effect human safety or mission success) using results from M&S, and establish a common set of terms and a uniform way for M&S practitioners to communicate the credibility of M&S. Figure 1 depicts these important drivers. Columbia Accident Investigation Board (CAIB) A Renewed Commitment to Excellence (Diaz Report) Return-to-Flight Task Group (Stafford-Covey Report) M&S VV&A Team Initiated October 2005 NASA Chief Engineer Memo, Sept. 2006 4 http://www.nasa.gov/home/hqnews/2005/aug/HQ_m05138_StaffordCovey.html Figure 1: Drivers for the NASA Modeling and Simulation Standard